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ABSTRACT 

This paper presents findings of a study that sought 
to delineate subtopics within school-effects research (SER) . After 
conducting a literature review and determining initial categories, a 
survey was conducted of 37 authors who were judged to have made the 
most significant contribution to SER within the last decade. A total 
of 28 authors responded. A second, revised survey was then sent to 
198 SER researchers and practitioners, which asked participants to 
rank 12 SER subtopics. A total of 142 responses were received, a 72 
percent response rate. Three types of individuals interested in SER 
were identified — scientists, pragmatists, and humanists. These three 
types appear to be roughly aligned with the three strands of 
SER — educational production-function studies, "effective 11 schools 
studies, and school-improvement studies. All respondents rated school 
change and school effects as important subtopics. However, the 
scientists were most concerned with methodological issues, whereas 
the humanists and pragmatists ranked teacher-school interaction and 
context issues highly. An implication is that by delineating 
subtopics of SER and the critical issues within them, the field of 
school effects can be advanced. Seven tables and one figure are 
included. (LMI) 
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More Clearly Defining the Field: A Survey 
of Subtopics in School Effects Research 



The field of school effects research (SER) is now over 25 
years old, if one considers the Coleman Report of 1966 as the 
first major study in the area. While there have numerous 
reviews of SER (i.e., Good i Br ophy ,1986; Levine & 
Lezotte, 1990; Purkey & Smith, 1983) , there is no widely 
accepted delineation of subtopics within this burgeoning 
literature. The current report is based on a two-phase survey 
of "experts" in the area with regard to their perceptions of 
the existence and importance of subtopics within SER. 

Subtopics Within SER: Why Haven't They Been Delineated? 

SER grew out of a response to the Coleman et al (196*) 
conclusion that schools make little difference in the lives of 
their students, but has proceeded to encompass a wide array of 
subtopics in the intervening 25+ years. SER in this report 
includes three major strands: educational production function 
studies that attempt to relate school inputs to school 
outcomes using increasingly sophisticated mathematical models; 
"effective schools" studies that describe characteristics of 
differentially effective schools using the outlier and case 
study approaches ; and school improvement studies that 
document the implementation and success of school change 
efforts. 



The inclusion of three such broad areas undex the rubric 
of school effects may* be surprising to some, but that is in 
fact the way a number of writers have envisioned the field. 
For instance, Good and Brophy's influential 1986 article in 
the Third Handbook of Research on Teaching entitled "School 
Effects" reviewed all three strands when they; (1) traced the 
roots of the area to Coleman's (1966) study, which utilized 
the educational production function methodology and analysis 
strategies; (2) focused on several case studies and outlier 
studies in the "effective schools" tradition; and (3) included 
reviews of the most important studies of school improvement at 
that time. 

One respondent referred to these three strands and their 
current relationships to one another: 

"Bringing together the methods at both extremes into 
a literature that is interpretable by the practitioner is an 
important issue today. We have the rigor of HLM and the 
richness of ethnographies, but school improvement models are 
not being developed based on the findings of either. Rather, 
the more easily understood 'correlates' form the basis for 
improvement models" (Respondent ft 23, first round survey). 

Despite the wide array of specific subtopics within each 

of these broad strands, there is no conventional taxonomy of 

SER subareas, although researchers often write about the 

context of school effects (e.g., Hallinger & Murphy, 1986; 

Teddlie & Stringfield, 1985, 1993; Wimpelberg,Teddlie & 

Stringfield, 1989 ), the stability of school effects (e.g., 

Mandeville & Anderson, 1987 ; Nutall,etal, 1989 ) , the magnitude 




of school effects (e.g., Brookover , etal , 1979 ; Reynolds, 1992) , 
and so forth. 

The are several reasons for this failure to delineate 
subtopics: (1) the fact that researchers in each of the 
different SER strands noted above have seldom worked together 
and in some cases see one another as competitors; (2) a 
reluctance to focus on SER, since school improvement is the 
raison d'etre for the field (not th^/ development of a 
"science 11 of school effects) according to many practitioners 
in the area ; (3) the dearth of school effectiveness/school 
improvement courses at the graduate level in both the USA and 
Europe, which (if they existed) would require professors to 
organize the literature into generally recognized subareas ; 
(4) residue from the severe criticism of the "effective 
schools" literature in the USA during the mid-1980 *s, which 
led to the labeling of the work as "reform posing as science" 
(i.e., Ralph & Fennessey,1983 ) and of the researchers as 
"shamans" (Rowan, 1984) ; and (5) the late development of 
certain important normative institutions in the field such as 
the AERA SIG (1984), the first annual meeting exclusively for 
those interested in the field (1988) and a journal devoted 
exclusively to the field (1990), These institutions were 
established 18-24 years after the Coleman Report. 

There is evidence that now is a good time to 
differentiate both subtopics and issues within each of those 
subtopics, which may then prove heuristic in generating 
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studies designed to probe more deeply the realities of srhool 
effects. Evidence for the maturity of the field include: 
(l)the growing body of research subsumed under school effects 
-(i.e., a recent ERIC using school effects as the key word and 
going back only 5 years yielded over 300 articles) ? (2) the 
emergence of an international journal ( School Effectiveness 
and School Improvement or SESI ) ,as well as the International 
Congress for School Effectiveness and Improvement (ICSEI), 
which will hold its seventh annual meeting in 1994 ; (3) the 
fact that SER is now overlapping in terms of research agendas 
with other, more established areas such as teacher effects 
(e.g. Kirby,1993; Teddlie, Kirby & Stringfield, 1989); and (4) 
the fact that researchers are now reporting replicable results 
in several areas such as the context, stability and magnitude 
of school effects. 

Research Questions and Methodology 

There were several research questions that guided this 
exploratory , primarily descriptive study: (1) What major SER 
subtopics can be delineated through a literature review? (2) 
What importance do experts in the field assign to each of the 
subtopics derived from the literature review? (3) What other 
areas do experts believe should be included in a taxonomy of 
important subtopics? (4) How would another group of experts 
rank a list of subtopics revised on the basis of answers to 



questions #2 and #3 above? (5) Do different groups of 
respondents respond differentially when rank ordering the 
subtopics? 

order to answer these questions , we undertook a 
multistage project including the fallowing phases: 

(1) Phase L Literature Review The literature review was 
undertaken to develop a set of categories for the first round 
of the study and to identify a sample of experts (researchers 
and practitioners),, who would serve as respondents. Sources 
for the literature review included an ERIC search using school 
effects and school effectiveness as the keywords , the 
literature reviews noted above plus others , all issues of 
SESI, the published proceedings of ICSEI and the catalogs of 
the last nine AERA meetings (starting with 1984 when the SIG 
was formed) using school effectiveness or school improvement 
as keywords. The ERIC search alone listed around 3000 
references , so we included only those published in the last 
five years. The final bibliography from these sources had 
400+ references , which were the most relevant from the much 
larger total list that emerged. 

(2) Phase 2. Determination of Initial Categories Using 
the titles of the references and their abstracts as text, we 
(the principal investigator and two graduate assistants) used 
the constant comparative method to arrive at preliminary SER 
subtopics (Lincoln & Cuba, 1985). This process was automated 
using Qualpro (Blackman, 1991 ) , a text database manager. Using 



three separate coders to increase the reliability/validity of 
the initial categorization scheme (i.e., Patton,1990) , a list 
of 17 distinct subtopics emerged. These subtopics were then 
used in the first round survey, which will be described below. 
The list of 17 subtopics together with brief descriptors is 
found in Appendix 1. 

(3) Phase 3. Development of First Round Survey and 
Sample 

The first survey consisted of two parts: a closed-ended 
section asking respondents to indicate the importance of the 
17 subtopics derived from Phase 2 on five point scales; and • a 
series of seven open-ended questions asking participants what 
are the most important current issues in SER, the most 
important methodological issues in SER, the current 
strengths /weaknesses of the research area, etc. 

From the 400+ references derived in Phase 1, we made a 
list of authors and then selected some 150 authors to be 
included in our sampling frame for the first survey. Selection 
criteria included number of references and importance of 
contribution to SER. These authors were also to be included in 
the sampling frame for the second round survey. 

Our final sample for the first survey included the 44 
authors who were consensus choices by three judges (the 
principal investigator and two graduate assistants) to have 
made the most significant contribution to SER over the past 
ten years. As with many aspects of this study, this selection 

6 

8 



procedure for the first round survey was subjective, but such 
is the nature of exploratory work in any field. Our intention 
was to include a more broad based sample for the second round 
survey . 

(4) Phase 4. Development of Second Round Survey and 
Sample Based on results from the first ' round survey, a 
revised list of subtopics was constructed deleting some with 
low ratings and adding others suggested from the open-ended 
responses. More information on this process will be included 
in the Results section. The second round survey consisted of 
asking respondents to rank order 12 SER subtopics from most to 
least important. A copy of the second round survey is located 
in Appendix 2. 

The expanded sample for the second round survey included: 
those respondents from the first round who responded to the 
first survey; those authors who were designated as significant 
contributors to SER, but were not included in the first round 
survey; and a list of practitioners sampled from three sources 
(AERA SIG list, SESI subscriber list, National Center for 
Effective Schools Research and Development membership list). 
The final sample frame consisted of 225 individuals. After 
deleting names of individuals no longer active in the field or 
for whom we did not have a current address, our final sample 
consisted of 209 individuals. 

7 9 
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Results 

Results from First Survey Descriptive Statistics from First 
Survey 

First round surveys were mailed to the 44 respondents 
identified in the previous section. Seven surveys were 
returned as undeliverable or were returned by respondents who 
refused to reply. Of the 37 remaining valid members from the 
first survey study, 19 returned usable completed 
questionnaires on the first mailout. A second mailout resulted 
in nine more responses, for a total of 28 responses. The total 
response rate of 77% (28 of 37) was considered adequate for 
this survey. 

Table 1 presents the means and standard deviations (sds) 
for the 17 response categories. (Refer to Appendix 1 for a 
listing and brief definitions of all 17 categories). Responses 
ranged from one (not important at all) to five (extremely 
important). The first notable result is the restricted range 
of responses to the five point scales. The lowest mean score 
was 3.5, while the highest was 4.6. Thus these experts availed 
themselves of a range of only 1.1 points on a 5 point scale. 
As one respondent noted, "They're all important". For this 
reason, we decided to use rank order scales for the second 
round survey. 

As noted in Table 1, the subtopic with the highest 
average score was "The Interaction of Teacher and School 
Effects", denoted as Teacher-School Interaction. Those 
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subtopics with the xowest mean values were "Definitions of 
School Effectiveness" (denoted as Definitions in Table 1) and 
"Parental Role in School Effects" (denoted as Parental Role in 
-Table 1). Seven of the subtopics had mean scores at 3.63 or 
lower , while 10 of the subtopics had mean scores of 3.7 or 
higher. 

The subtopic with the largest sd was "External School 
Improvement" , which was also one of the items with a low 
overall rating (3.59). The subtopic with the highest overall 
rating (Teacher-School Interaction) also had the smallest sd 
(0.57) . 

Revision of Subtopic List Based on First Study Results 
As noted above, a major purpose of the first survey was 
to revise the list of subtopics based on expert opinion. 
This process was two-fold: first, those subtopics with low 
mean ratings on the survey were to be eliminated; second, the 
open-ended responses were to be used to generate new and to 
revise existing subtopics. 

The following procedure was used to produce the list of 
subtopics used in the second round study: 

(1) Those subtopics with the lowest overall mean scores 
were eliminated from the list. The seven subtopics with mean 
scores of 3.63 or lower were dropped, although some of them 
were later added back to the list in different form, as noted 
below. One eliminated subtopic was "The Correlates of School 
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Effectiveness" , which several respondents indicated was of 
great historical importance , but lacked current relevance, 

( 2 ) The subtopic "The Multilevel Nature of School 
Effects" was eliminated , since several respondents said it was 
redundant to other subtopics (such as, Teacher-School 
Interaction or "District Effects upon School Effects"), 

(3) Several respondents suggested that we substitute "The 
Role of Leadership in School Effects" for "The Role of the 
Principal in School Effects", which we did. 

(4) Several respondents also suggested that we combine 
"The Consistency of School Effects across Measures" with "The 
Stability of School Effects across Time". We combined these 
categories in the second round survey. 

(5) Some respondents suggested that we expand the 
definition of the subtopic "District Effects upon School 
Effects" to include "State Effects" also. We did this, and we 
also expanded the brief description to include "the effects of 
units above the school, including school boards". 

(6) While "External School Improvement" and "Naturally 
Occurring School Change " were eliminated due to low mean 
scores, respondents suggested that we include some subtopic 
that addressed the issue of school change. We included tne 
subtopic "The Relationship of School Effects to School Change 
(e,g., school improvement, school restructuring, site-based 
management)" to address this issue. 



(7) Several respondents suggested that we add two new 
subtopics: "School Effects and Curriculum Issues" and 
"International Comparative Studies of School Effects". While 
neither of these topics showed up prominently in our 
literature review, respondents felt that these topics will 
become increasingly important in the future. 

For instance, one respondent spoke of curriculum issues 
as follows: 

"I feel it is critical to look at the relationship 
between school effectiveness and the curriculum (broadly 
defined as everything happening within the classroom itself) 
because ultimately the focus is on the effect or impact of 
these other variables on the learning - how it ultimately 
impacts kids" (Respondent # 110, second round survey). 

All of these changes, which are summarized in Figure 1, 

were incorporated into the revised subtopic list found in 

Appendix 2. As might be expected, some of the respondents were 

not satisfied with the revised list of subtopics, as 

illustrated by the following quote: 

"Some of the items on the reverse side are vague or 
combinations of independent elements. .. .1 think this dooms the 
findings of this research to be ambiguous" (Respondent # 187, 
second round survey.) 

We agree, but believe that an "ambiguous" beginning to 
delineating SER is better than no start at all. 

The qualitative analysis of open-ended responses was used 
in revising the subtopic list. These responses contained a 
wealth of information on several topics, which cannot be 
presented in this paper due to time constraints. Further 
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analysis of this qualitative data will be presented in a 
future paper. 

Results from Second Survey Overall Rankings for the Twelve 
Subtopics 

Second round surveys were mailed to the 209 respondents 
identified in the Methods section. Eleven surveys were 
returned as undeliverable or were returned by respondents who 
refused to reply. Of the 198 remaining members on the second 
survey list, 142 returned usable, completed questionnaires. 
This response rate of 72% for a one time mailout is quite 
high, indicating considerable enthusiasm for the survey. 

Of the 198 members of the final sampling frame, 28% (55) 
were not from the USA. The response rate at the time of the 
report was 65% (36/55) for the non-USA respondents and 74% 
(106/143) for the USA respondents. 

Borg and Gall (1989) reported a synthesis of the survey 
literature indicating that response rates for one time 
mailouts averaged 48%, while three mailouts were required to 
attain the greater than 70% rate that we got on our first try. 
Due to this high response rate and time constraints, we 
decided to not undertake a second mailout. (Two more 
respondents returned their questionnaires after the data 
analysis reported here was completed, raising the response 
rate to 73%) . 
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Table 2 presents the median and interquartile ranges for 
responses to the SER subtopics. This table also presents means 
and sd s for each of the twelve subtopics, based on an analysis 
that treated the data as if it were parametric- The results 
:rom the two analyses were nearly identical in that: 

(1) The median and mean results reported the exact same 
ordering of SER subtopics in perceived importance ranging from 
"The Relationship of School Effects to School Change" (denoted 
School Change in Table 2), which was ranked most important, to 
"International Comparative Studies of School Effects" (denoted 
International Studies) which was ranked least important. 

(2) The interquartile ranges and standard deviations 
revealed nearly identical scores also, ranging from School 
Change which had the least variance in responses to "the 
Existence and Magnitude of School Effects" (denoted 
Existence/Magnitude Issues in Table 2), which manifested the 
most variance. 

Due to the high degree of similarity between the 
parametric and nonparametric descriptive data, we decided to 
employ inferential parametric techniques when further 
analyzing the data. As noted by Harris ( 1985 ) : 

. . .we have such strong mathematical and empirical 
evidence of the robustness of statistical procedures under 
violation of normality or homogeneity of variance assumptions 
that the burden of proof must be presumed to be on the 
shoulders of those who claim that a particular set of data can 



be analyzed only through 1 nonparametric ' statistical 
techniques (p. 326) . 

The "burden of proof" was not demonstrated in this 
dataset , 

which has descriptive nonparametric statistics almost 
identical to those generated by re-analysis of the data using 
descriptive parametric statistics. Thus, we used parametric 
statistics, such as factor analysis and multivariate analysis 
of variance (MANOVA), in further analyses. 

Factor Analysis of Responses to Second Survey 

Factor analytic work employing principal components 
analysis, followed by a varimax rotation, resulted in a five 
factor solution (if the eigenvalue greater than or equal to 
1.00 criteria was used) or a three factor solution (if a scree 
test was utilized). We decided to utilize a one factor 
solution for the following reasons: 

(1) The five factor solution had two uninterpretable 

factors • 

(2) The three factor solution had factors which 
included items that were highly positively and highly 
negatively loaded. Further perusal of these three factors 
indicated two distinct patterns of response, one of which 
seemed to imply a humanistic, applied research, school change 
orientation, while the other suggested a more scientific, 
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basic research, theoretical orientation. Table 3 contains a 
summary of the factor loadings from the three factor solution. 

(3) A one factor solution was ideal for further 
exploring these two orientations in a singular response 
pattern. The one factor solution assigns a valence to each 
subtopic, and since there is just one pattern of loadings to 
examine, these loadings can then be grouped into two 
categories (one positive, one negative). We surmised that if 
the subtopics within each valence category grouped together 
based on the different orientations suggested in (2) above, 
then there would be empirical support for the conjecture that 
some responders are ranking subtopics in one manner, while 
others are ranking them differentially. 

The data in Table 4 appear to confirm the existence of 
these two distinct response patterns. Those subtopics with 
positive valences included: (1) School Change; (2) Teacher- 
School Interaction; (3) Role of Leadership; (4) Curriculum 
Issues (School Effects and Curriculum Issues); (5) Context 
Issues (The Context of School Effects) ;and (6) District-State 
Effects [District and State Effects Upon School 
Effects ( i.e. ,the effects of units above the school)]. 

Those subtopics with negative valences included: (1) 
Methodological Issues [in SER (e,g., validity, reliability, 
generalizability, etc.)]; (2)Variance Issues (The Variance of 
Behaviors within Differentially Effective Schools); (3) 
Stability Issues (The Stability of School Effects across Time 



and Consistency of School Effects across Measures ) ; ( 4 ) 
Existence/Magnitude Issues; (5) Theory (the Role of Theory in 
SER); and (6) International Studies* 

Perusal of these two sets of subtopics confirm earlier 
speculation arising from the results of the three factor 
solution. Respondents giving high ranks to subtopics with 
positive valences are probably more interested in applied 
studies of school effects that have direct implications for 
school change processes. We might call these SER researchers 
"humanists 11 to use the terminology of Cronbach(1982) • 

On the other hand, those respondents who gave high ranks 
to subtopics with negative valences are more interested in 
theoretically driven studies of the basic school effects 
processes. As one scientist pithily put it, "Theory is first, 
methodology is second, more specific issues third" 
(Respondent # 24, second round survey), Cronbach would call 
these individuals "scientists" and contrast their orientation 
with that of the "humanists" previously described. 

Results of Comparisons between Humanists and Scientists 

The data in Table 5 present the results of univariate 
ANOVA F tests in which the responses of these two groups 
( humanists , scientists ) were compared . Group assignment 
resulted from multiplying each respondent's ranks by the 
factor loading for each of the subtopics. Then, respondents 1 
total scores were arrayed from the most positive to the most 



negative. A median split was performed and approximately half 
(72) of the respondents were classified as humanists , while 
the others (70) were classified as scientists. 

The overall multivariate F for the comparison between the 
two groups on their rankings was highly significant [ (F 
12, 129) =24.53 , £<.0001], as was expected. Two of the most 
highly significant univariate differences between the 
humanists and the scientists came on the importance of School 
Change and of Methodological Issues. As indicated in Table 5, 
which also compares the order of the rankings for the two 
groups, the humanists gave School Change an average rank of 
2.94 (their highest overall rank), while the scientists gave 
it a rank of only 5.57. On the other hand, the scientists 
gave Methodological Issues their highest average rank of 4.39, 
while the humanists gave it a much lower average rank of 8.11. 

Tables 6 compares the rankings given by the total group, 
the humanists and the scientists. The rankings given by the 
humanists more closely parallel those given by the total group 
than do the rankings given by the scientists. Since each group 
was equally represented in the total sample, it is curious 
that one group's responses would be more representative of the 
total sample. The reason for this phenomenon lies in the 
variances of responses from the two groups. As indicated in 
Table 5, there is a range of about seven ranks between the 
humanists' highest average ranking (2.94 for School Change) 
and their lowest average ranking (9.96 for International 

17 

19 



Ctudies). On tne other hand, the range for the scientists is 
much smaller (4.74) , going from a high rank of 4.39 for 
Methodological Issues to 9.13 for District-State Effects. 

The standard deviations for the two group's rankings 
confirms the differences in variations of responses , with the 
humanists having a sd of 1.97, while the sd for the scientists 
was only 1.39. Scientists appear to consider many topics 
important, including some of those associated with the 
humanistic orientation. On the other hand, humanists 
definitely prefer those subtopics associated with School 
Change, and gave low rankings to almost all the scientific 
orientation subtopics. 

Results of Comparisons among Three Groups 

Another way to look at the classification of respondents 
in the second round survey would involve a tripartite split: 
(1) humanists - those individuals scoring in the upper one- 
third of the continuum described in the previous section; (2) 
pragmatists - those individuals scoring in the middle one- 
third of the continuum; and (3) scientists - those individuals 
scoring in the lower third of the continuum. The mid-range 
classification has some ecological validity since there are 
many individuals who consider themselves to be both 
practitioners and researchers. We have decided to call these 
individuals "pragmatists" , to use a derivation of the term 
Howe (1988) employed in describing a new theoretical 
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orientation (pragmatism) that he said combined aspects of both 
positivism (the theoretical position associated with the 
scientific orientation) and post-positivism (the theoretical 
position associated with the humanist orientation). While 
"fence-sitters' 1 or "middle-of-the-roaders" might be more 
descriptive terms for these individuals , we will use the less 
pejorative pragmatist label here. 

Data presented in Table 7 indicate significant 
differences between the three groups on average rankings for 
10 of the 12 subtopics. The overall multivariate effect was 
highly significant [F(24, 254) =28.64, £<.0001]. On all 
rankings, the score of the middle-of-the roaders was 
intermediate between that of the scientists and the humanists. 

Post-hoc tests revealed that for four of the subtopics, 
the three groups differed significantly from each other. For 
four of the subtopics the responses of the scientists differed 
significantly from those of the other two groups. For those 
subtopics, pragmatists responded similarly to humanists. 

This is particularly interesting on two subtopics: 
School Change and Existence/Magnitude Issues. The pragmatists 
believe School Change is the most important subtopic in SER, 
and rank Existence/Magnitude Issues low in importance. Several 
of the respondents in the humanist and pragmatist groups 
indicated that Existence/Magnitude Issues had historical 
importance, but were no longer that important, since the 
existence of school effects were now well established. As one 
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respondent noted, "School effects always exist - change is the 
issue" (Respondent # 37, first round survey). Scientists, on 
the other hand, believe that the study of the size of school 
effects was still very important* 

Discussion 

This study confirms the "schizophrenic" nature of that 
group of individuals who are interested in SER. According to 
their own response patterns there are at least two types of 
individuals interested in SER: those we have called the 
humanists and the scientists. Responses indicate that there 
may even be a third group, who make responses somewhere 
between the two groups. 

From a methodological perspective, it is reassuring that 
we have all three groups represented in our final sample. This 
means that we have sampled the full range of those individuals 
interested in SER. Returning to a point from the 
introduction, if we believe that there are three distinct 
strands of SER ( educational production function studies , 
"effective" schools studies, school improvement studies), then 
our "types" of respondents ( scientists, pragmatists, humanists) 
roughly align with each of those orientations. 

These analyses also indicate that both the AERA SIG and 
the journal are aptly named (School Effectiveness and School 
Improvement) since respondents (to at least this poll) place a 
heavy emphasis on school change, as well as school effects. 
Even the scientists rated school change as their fourth most 



important area. On the other hand f the importance of 
Methodological Issues (the number one concern of scientists) 
was ranked fifth by pragmatists and seventh by humanists. 
Apparently, only the scientists are really interested in 
theory (ranked third by them) , since the pragmatists and 
humanists ranked it ninth and eighth respectively. 

The importance of the Teacher-School Interaction and of 
Context Issues to both the pragmatists and humanists is 
noteworthy. These are two areas where replicable results have 
been uncovered over the past five years. 

As one respondent noted: 

"Probably the most important issues are the context 
of school effects and the interaction of teacher and school 
effects.... Both are areas in which school effects can be 
considered 'more scientific 1 since studies in these areas 
should continue to yield interesting and replicable findings 11 
(Respondent # 41, first round survey). 

It is interesting that the scientists, who are the most 

likely to conduct SER, do not rate these two areas that highly 

(Teacher-School Interaction was rated eighth and Context 

Issues fifth by them) . The scientists appear to be 

preoccupied with more traditional SER subtopics such as the 

consistency, stability, existence and magnitude of school 

effects • 

Some respondents didn't consider these surveys very 

valuable, even though they gracefully completed them. As one 

humanist said: 

"Future research should focus on school reform and 
change. Why in the world is a rank ordering useful? What 
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decisions will be made on faculty prioritizing their 
perceptions?" (Respondent-* 37 , second round survey). 

These are fair questions , deserving appropriate response. 

First, we hope that this research will inform "faculty" and 

other respondents that there are others doing SER in very 

similar , and very dissimilar manners. We need to remain aware 

that there are at least three distinct traditions within our 

field , and that each group of researchers or practitioners 

have their own values and agendas. The first step in more 

closely working together is to understand our differences and 

look for commonalities in future endeavors to advance the 

field. 

Secondly , presenting a list of subtopics in and of itself 
should prove educational to many interested in SER. Fcr 
instance , as noted above , those with scientific orientations 
rank Context Issues and Teacher-School Interactions low, 
although studies published recently in these areas are 
certainly "scientific" in terms of rigor of design and 
replicability of results. 

Similarly, humanists should be aware that many interested 
in SER are still actively probing the existence and magnitude 
of certain variables using increasingly sophisticated methods, 
which should yield more definitive answers in this area. By 
closely reading these "scientific" studies, humanists and 
pragmatists may find information of use in their school change 
efforts. 
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Finally, the emergence of important subtopics will 
hopefully stimulate researchers to closely review recent 
findings in all these areas. Optimally the long-term effect 
of a list of prioritized major SER subtopics would be to 
stimulate coherent chains of research , each building on the 
latest findings. For instance , the study of the effect of the 
SES of a school's student population on that school! s process 
for becoming and maintaining an effective school has yielded a 
series of studies from the exploratory and primarily 
descriptive (Hallinger & Murphy , 1986; Teddlie & Stringfield, 
1985) to the confirmatory , utilizing a prior hypotheses (Evans 
& Teddlie, 1993; 'fleck, 1992). 

As Creemers and Reynolds (1990) noted, "In the early 
stages of a discipline there is much historical evidence that 
a problem solving approach is that which is most likely to 
generate intellectual advances" (p. 3) • By delineating 
subtopics, and hopefully critical issues within those 
subtopics, the field of school effects should be advanced. At 
the very least, such a list of subtopics ( "endorsed" by 
experts in the field) should encourage someone to write a 
textbook or teach a graduate level course in the area. 
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Table 1 



Respondents 1 Ratings of Importance 
of Subtopics from First Round Survey 



SUBTOPIC 
DEVIATION 

Teacher-School Interaction 
Context Issues 
Role of Principal 
Multilevel Issues 
Methodological Issues 
Variance Issues 
Stability Issues 
District Effects 
Existence /Magnitude Issues 
Theory 

Consistency Issues 
Correlates 

Teacher Induction/Social* 
External School Improvement 
Naturally Occurring Sch.Imp. 
Parental Role 
Definitions 



MEAN STANDARD 



4.59 


0.57 


4.22 


0.75 


4.22 


0.80 


4.15 


1.06 


3.93 


1.07 


3.93 


1.10 


3.89 


1.05 


3.78 


1.18 


3.78 


1.37 


3.70 


1.17 


3.63 


1.14 


3.63 


1.14 


3.59 


0.84 


3.59 


1.42 


3.58 


1.17 


3.52 


0.94 


3.48 


1.28 



Note , The data in this table are based on questionnaires 
returned from 28 respondents (77% of sample) . Response 
categories for each subtopic range from 1 (not important at 
all) to 5 (very important). The names of the subtopics found 
in this table are shortened from those given on the survey, 
which is located in Appendix 1. 
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Figure 1 

Revision of Subtopics for Round Two 
Survey Based on Responses to Round One 

ROUND ONE SUBTOPICS ROUND TWO SUBTOPICS 

( l) Teacher-School Interaction > Same (1) 

( 2) Context Issues > Same < 2) 

( 3) Role of the Principal > Renamed Role of 

Leadership (3) 

( 4) Multilevel Issues > Eliminated due 

to Redundancy 
with other 
Subtopics 

( 5) Methodological Issues > Same as (4) 

( 6) Variance Issues > Same as (5) 

( 7 ) stability Issues > Combined with 

Consistency and 
Renamed 
Stability and 
Consistency 
Issues 

( 8) District Effects > Expanded to 

Include State 
Effects and 
Renamed 
District State 
Effects 

( 9) Existence /Magnitude Issues > Same (8) 

(10) Theory > SaITie O) 

(11) Consistency Issues > Combined with 

Stability 
Issues 

(12) Correlates > Eliminated due 

to low ratings 

(13) Teacher Induction/Socialization > Eliminated due 

to low ratings 
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Table 2 

Descriptive Summary of Respondents' Rankings 
of Subtopics from Second Round Survey 



INTERQUARTILE STANDARD 



SUBTOPIC 


MEDIAN 


MEAN 


RANGE 


DEVIATION 


scnool cnange 






2-5, 


4 


ranks 


7 RQ 


Teacner-scnooi 
Interactions 


4 


A 1 A 

4 . /4 


2-6, 


5 


ranks 


*5 Q"i 


Context Issues 


5 


5.51 


3-7, 


5 


ranks 


2.94 


Role of Leader- 
snip 


5 


5.94 


3-8, 


6 


ranks 


3.25 


Stability/ 
consistency 


6 


6.04 


4-8, 


5 


ranks 


2.93 


Curriculum 
Issues 


6 


6.14 


3-8, 


6 


ranks 


3.19 


Methodological 
Issues 


6 


6.23 


3-8, 


6 


ranks 


3.37 


Variance Issues 


6 


6.61 


4-8, 


5 


ranks 


3.04 


Theory 


7 


6.70 


4-9, 


6 


ranks 


3.63 


Existence/ 

Magnitude 

Issues 


7 


7.08 


4-9, 


6 


ranks 


3.79 


District/State 
Effects 


8 


7.83 


6-9, 


4 


ranks 


2.76 


International 
Studies 


10 


9.27 


7-11, 


5 


ranks 


3.27 



Note * The data in this table are based on questionnaires 
returned from 142 respondents (72% of sample). Respondents 
ranked subtopics from 1 (most important) to 12 (least important). 
If a respondent did not rank order all items , the remaining items 
were assigned a rank that was the average of the remaining ranks. 
For instance , if a respondent ranked nine items , the remaining 
three items were ranked 11 , which is the average of the remaining 
three ranks (10,11,12). 
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Figure 1 (Continued) 
(14) External School Improvement > 



Relabeled 
School Change 



(10) Due to 
Respondents ' 
Comments 



(15) Naturally Occurring School Improvement — > 



Relabeled 
School Change 
(10) Due to 
Respondents 1 
Comments 



(16) Parental Role 



(17) Definitions 



Eliminated due 
to low ratings 

Eliminated due 
to low ratings 



Not on Original List 



Curriculum 



Issues 



(11) 
due to 



Added 
Respondents 1 
Comments 



Not on Original List 



International 
Studies (12) 
Added due to 
Respondents 1 
Comments 
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Table 3 

Subtopic Loading from Factor Analysis of 
Responses of Second Round Participants: 
Three Factor Solution 



SUBTOPIC 

School Change 

Teacher-School 
Interaction 

Context Issues 

Role of Leadership 

Stability/ Consistency 
Issues 

Curriculum Issues 
Methodological Issues 
Variance Issues 
Theory 

Existence/Magnitude 
Issues 

District/State Effects 
International Studies 



FACTOR 1 
.05 
-.58* 

-.14 
-.11 
.66* 

-.52* 
.17 
.18 
.06 
.75* 

-.16 
-.34 



FACTOR 2 
-.01 
.06 

-.41* 
-.67* 
.18 

-.13 
.27 

-.24 
.56* 

-.11 

-.34 
.66* 



FACTOR 3 
.82* 
.03 

-.34 
.16 
-.14 

.18 
-.63* 
-.37* 
-.04 

.05 

.52* 
-.09 



Note . These results are from a principal components factor 
analysis of the second round dataset, followed by a varimax 
rotation. Data from 142 respondents were included in this 
analysis. Factor loadings greater in magnitude than + .35 are 
asterisked. 
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Table 4 



Subtopic Loading from Factor Analysis of 
Responses of Second Round Participants: 
One Factor Solution 



SUBTOPIC 

School Change 

Teacher-School 
Interaction 

Context Issues 

Role of Leadership 

Stability /Consistency 
Issues 

Curriculum Issues 
Methodological Issues 
Variance Issues 
Theory 

Existence /Magnitude 
Issues 

District/State Effects 
International Studies 



FACTOR 1 

• 50* 

• 32 

.07 

• 50* 
-.56* 

• 48* 
-•65* 
-.21 
-.36* 
-.34 

• 60* 
-.20 



ORIENTATION 
Humanistic 
Humanistic 

Humanistic 
Humanistic 
Scientific 

Humanistic 
Scientific 
Scientific 
Scientific 
Scientific 

Humanistic 
Scientific 



Note . These results are from a principal components factor 
analysis of the second round dataset. Data from 142 respondents 
were included in this analysis • Factor loadings greater in 
magnitude than + .35 are asterisked • 
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Table 5 



Average Rank Scores Given by 
Humanists and Scientists to Subtopics 



SUBTOPIC HUMANISTS' SCIENTISTS' F-VALUE 

AVERAGE RANK AVERAGE RANK 





MEAN 


RANK 


MEAN 


RANK 




scnool cnange 


z . y 4 


1 




A 
H 


"36 SO**** 


Interaction 


j • / D 


O 


^ 70 


6 

w 


17 . 27**** 


context issues 


c; ah 
D . 4 / 




J . J H 




0 02 


Role of Leadership 


4.74 


3 


7.11 


9 


21.56**** 


Stability/ 
Consistence Issues 


7.37 


8 


4.75 


2 


35.28**** 


Curriculum Issues 


4.89 


4 


7.36 


10 


25.00**** 


Methodological 
Issues 


8.11 


11 


4.39 


1 


62.23**** 


Variance Issues 


6.86 


7 


6.36 


8 


0.94 


Theory 


7.77 


9 


5.67 


5 


12.96*** 


Existence/Magnitude 
Issues 


8.10 


10 


6.08 


7 


10.73** 


District/State 
Effects 


6.50 


6 


9.13 


12 


41.09**** 


International 
Studies 


9.96 


12 


8.61 


11 


6.23* 



Note . Seventy- two of the respondents were classified as 
humanists, while 70 were classified as scientists. Respondents 
ranked subtopics from 1 (most important) to 12 (least important). 

* p < .05 

** p < .01 

*** p < .001 

**** p < .0001 



Table 6 



Comparison of Rankings for All Respondents , 
Humanists and Scientists 



SUBTOPIC ALL 


RESPONDENTS 


HUMANISTS 


scira 


School Change 


1 


1 


4 


Teacher-School 
Interaction 


2 


2 


€ 


Context Issues 


3 


5 


3 


Role of Leadership 


4 


3 


9 


St* afc i 1 i tv / C on s i stencv 
Issues 


5 


8 


2 


Curriculum Issues 


6 


4 


10 


Methodological Issues 


7 


11 


1 


Variance Issues 


8 


7 


8 


Theory 


9 


9 


5 


Existence/Magnitude 
Issues 


10 


10 


7 


District/State Effects 


11 


6 


12 


International Studies 


12 


12 


11 



Note . There were 142 respondents , of which 72 were classified as 
humanists and 70 as scientists. Respondents ranked subtopics 
from 1 (most important) to 12 (least important). A rank of 1 on 
this table means the respondents 1 average rank for that subtopic 
was the lowest (meaning it was the most important subtopic for 
that group) . 
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Table 7 

Average Rank Scores Given by 
Humanists, Pragmatist and Scientist to Subtopics 



SUBTOPICS HUMANISTS' PRAGMATISTS ' SCIENTISTS' 

MEAN RANK MEAN RANK MEAN RANK F -VALUE 



School Change 


2 . 


83 


1 


3 . 


91 


1 


6 . 


06 


7 


19 . 


£ ^ * * * * 


leacner-scnooi 


■a 


n q 




A 

4 . 


li 


z 


c 

O . 


08 


0 


8 „ 




Interaction 
























Context Issues 


5, 


21 


5 


5. 


59 


3 


5. 


72 


5 


0. 


40 


Role of 
























Leadership 


3. 


96 


3 


6. 


20 


6 


7. 


69 


10 


20. 


37* *** 


stability/ Consi 


8 . 


06 


9 


5 . 


96 


4 


4 . 


10 


2 


31 . 


1 3 * * * * 


stency Issues 
























Curriculum 


4. 


63 


4 


6. 


22 


7 


7. 


58 


9 


11. 


94 * * * * 


Issues 
























Methodological 
























Issues 


S. 


69 


11 


6. 


13 


5 


3. 


88 


1 


37. 


53* * * * 


Variance Issues 


7. 


52 


7 


6. 


52 


8 


5. 


77 


6 


4. 


17* 


Theory 


7. 


71 


8 


7. 


17 


9 


5. 


25 


3 


6. 


56** 


Existence/ 
























Magnitude 


8. 


29 


10 


7. 


63 


10 


5. 


33 


4 


8. 


93* * * 


Issues 
























District/ 


5. 


83 


6 


8. 


22 


11 


9. 


46 


12 


30. 


07 * * * * 


State Effects 
























International 


10. 


08 


12 


8. 


96 


12 


8. 


77 


11 


2. 


30 


Studies 

























Note . Forty-eight of the respondents were classified as 
humanists, 46 as pragmatists and 48 as scientists. 
Repondents ranked subtopics from 1 (most important) to 12 
( least important ) . 



* p<.05 

** p<.005 

*** p<.0005 

**** p<.0001 



